{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d27f1082-cd10-405e-9570-6f0e934bba8b",
   "metadata": {},
   "source": [
    "# LlamaParse JSON Mode + Multimodal RAG\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_cloud_services/blob/main/examples/parse/demo_json.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "This notebook shows you how to use LlamaParse JSON mode with LlamaIndex to build a simple multimodal RAG pipeline.\n",
    "\n",
    "Using JSON mode gives you back a list of json dictionaries, which contains both text and images. You can then download these images and use a multimodal model to extract information and index them.\n",
    "\n",
    "Status:\n",
    "| Last Executed | Version | State      |\n",
    "|---------------|---------|------------|\n",
    "| Aug-19-2025   | 0.6.61  | Maintained |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a004db48-8d3f-421c-915a-477692f71b90",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Define imports, env variables, global LLM/embedding models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc6a7a4b-b568-4db5-bcba-62f5c517ff3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index\n",
    "%pip install \"llama-index-core>=0.13.2<0.14.0\"\n",
    "%pip install \"llama-index-llms-anthropic>=0.8.4<0.9.0\"\n",
    "%pip install \"llama-index-embeddings-huggingface>=0.6.0<0.7.0\"\n",
    "%pip install llama-cloud-services"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0879301c-ff91-4431-941a-6c0ef7cd8fe2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# API access to llama-cloud\n",
    "os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"llx-...\"\n",
    "\n",
    "# Using Anthropic API for LLMs\n",
    "os.environ[\"ANTHROPIC_API_KEY\"] = \"sk-...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "391e2d95-5569-4d73-9f16-5b59d7326f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.anthropic import Anthropic\n",
    "\n",
    "llm = Anthropic(model=\"claude-4-sonnet-20250514\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "700f48e8-8b52-41f3-90f9-144d5fdd5c52",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/loganmarkewich/llama_parse/py/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = \"local:Qwen/Qwen3-Embedding-0.6B\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b411d2ee-3e6b-45b0-b532-4a8e3abcdea0",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "Let's load in the Uber 10Q report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c39d408f-e885-4940-85c7-b09ca3bc7cb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10q/uber_10q_march_2022.pdf' -O './uber_10q_march_2022.pdf'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2f42af8-afb3-4b3b-82d3-6b332fb38aa4",
   "metadata": {},
   "source": [
    "## Using LlamaParse in JSON Mode for PDF Reading\n",
    "\n",
    "We show you how to run LlamaParse in JSON mode for PDF reading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c9cd670-8229-4ad6-99a9-845bd82b7ec1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started parsing the file under job_id 33d93a46-1b43-4619-b4ff-0c272cbca4b3\n",
      ".."
     ]
    }
   ],
   "source": [
    "from llama_cloud_services import LlamaParse\n",
    "\n",
    "parser = LlamaParse(\n",
    "    parse_mode=\"parse_page_with_agent\",\n",
    "    model=\"openai-gpt-4-1-mini\",\n",
    "    high_res_ocr=True,\n",
    "    adaptive_long_table=True,\n",
    "    outlined_table_extraction=True,\n",
    "    output_tables_as_HTML=True,\n",
    ")\n",
    "\n",
    "result = await parser.aparse(\"./uber_10q_march_2022.pdf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "364a3276-d2db-4aee-9bc6-617ffd726d25",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_nodes = await result.aget_text_nodes(split_by_page=True)\n",
    "image_nodes = await result.aget_image_nodes(\n",
    "    include_screenshot_images=True,\n",
    "    include_object_images=False,\n",
    "    image_download_dir=\"./uber_10q_images\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fe2e911-0393-42e8-a233-65639cdbebc4",
   "metadata": {},
   "source": [
    "## Extract/Index images from image dicts\n",
    "\n",
    "Here we use a multimodal model to caption images and create text nodes for indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36012145-5521-4ddb-a53e-df9ebd1ca8dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.llms import ChatMessage, ImageBlock, TextBlock\n",
    "from llama_index.core.schema import ImageNode, TextNode\n",
    "from llama_index.llms.anthropic import Anthropic\n",
    "\n",
    "\n",
    "async def get_image_text_nodes(image_nodes: list[ImageNode]):\n",
    "    \"\"\"Extract out text from images using a multimodal model.\"\"\"\n",
    "    llm = Anthropic(model=\"claude-3-5-haiku-20241022\", max_tokens=300)\n",
    "    img_text_nodes = []\n",
    "    for image_node in image_nodes:\n",
    "        image_path = image_node.image_path\n",
    "        message = ChatMessage(\n",
    "            role=\"user\",\n",
    "            blocks=[\n",
    "                TextBlock(text=\"Describe the images as alt text\"),\n",
    "                ImageBlock(path=image_path),\n",
    "            ],\n",
    "        )\n",
    "        response = await llm.achat([message])\n",
    "        text_node = TextNode(\n",
    "            text=str(response.message.content), metadata={\"path\": image_path}\n",
    "        )\n",
    "        img_text_nodes.append(text_node)\n",
    "\n",
    "    return img_text_nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38f25045-6102-4920-9cd0-42b0ae6c872f",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_text_nodes = await get_image_text_nodes(image_nodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4683c97a-da06-408a-9fe9-7e3c0aceb77d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Alt text: United States Securities and Exchange Commission Form 10-Q for Uber Technologies, Inc., dated for the quarterly period ended March 31, 2022. The document shows company details including incorporation state (Delaware), address (1515 3rd Street, San Francisco), and indicates Uber is a large accelerated filer listed on the New York Stock Exchange with the trading symbol UBER.'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_text_nodes[0].get_content()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cfdf6db-381c-4e53-a0fb-e7670f75e0d5",
   "metadata": {},
   "source": [
    "## Build Index across image and text nodes\n",
    "\n",
    "Here we build a vector index across both text nodes and text nodes extracted from images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "939aec6c-064a-4319-b2dc-70cc4a304c06",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex\n",
    "\n",
    "index = VectorStoreIndex(text_nodes + image_text_nodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "529340d5-9319-4cdf-8ee1-bbd01ed00226",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81d7ff30-5a87-44da-880d-4b1f41434d90",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The bar graph titled 'Monthly Active Platform Consumers' shows the growth in platform users measured in millions from Q2 2020 to Q1 2022. The graph demonstrates a steady increase in the number of consumers using the platform, starting at 55 million users in Q2 2020 and rising to 115 million users in Q1 2022. The visualization displays notable growth between quarters, with the vertical axis representing the number of consumers in millions and the horizontal axis showing the quarterly progression over this two-year period.\n"
     ]
    }
   ],
   "source": [
    "# ask question over image!\n",
    "response = query_engine.query(\n",
    "    \"What does the bar graph titled 'Monthly Active Platform Consumers' show?\"\n",
    ")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4f14ad8-6bfd-49d9-b3d5-7215cf0e4ac1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the financial documents provided, I can identify some key risk factors for Uber, though the context is limited to specific pages:\n",
      "\n",
      "**Legal and Regulatory Risks:**\n",
      "- Driver classification issues pose significant business risks, as legal determinations about whether drivers are employees or independent contractors could substantially impact Uber's operations and cost structure.\n",
      "\n",
      "**Operational Risks:**\n",
      "- The company continues to report net losses, indicating ongoing profitability challenges across its business segments.\n",
      "\n",
      "**Business Model Risks:**\n",
      "- Uber operates across multiple segments (Mobility, Delivery, and Freight), which creates exposure to various market conditions and regulatory environments in different industries.\n",
      "\n",
      "**Geographic Concentration Risk:**\n",
      "- The company has operations across different geographic regions, which exposes it to varying regulatory frameworks, economic conditions, and competitive landscapes in different markets.\n",
      "\n",
      "However, the provided context appears to be from specific pages of financial reports that focus primarily on financial metrics and segment information. A complete assessment of Uber's risk factors would typically be found in the dedicated risk factors section of their SEC filings, which is not included in the available context.\n"
     ]
    }
   ],
   "source": [
    "# ask question over text!\n",
    "response = query_engine.query(\"What are the main risk factors for Uber?\")\n",
    "print(str(response))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
